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(54) Automatic generation of test drivers 



(57) A test driver generator is provided for generat- 
ing test drivers. The test driver generator receives test 
expressions designating execution sequences of test 
functions of software interfaces and corresponding at- 
tribute value specifications for the designated test func- 
tions' parameter attributes. Each test expression desig- 
nating a number of test functions to be executed in a 
certain sequence, and each corresponding attribute val- 



ue specification specifies selected attribute values of the 
test functions' parameter attributes. For each test ex- 
pression and corresponding attribute value specifica- 
tions of a software interface, the test driver generator 
in response, gene rates a test driver that can execute the 
specified test functions in the designated order with all 
combinations of the selected attribute values of the test 
functions' parameter attributes. 
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Descripti n 

The present invention relates to the field of software 
testing. 

More specifically, the present invention relates to 
automatic generation of test drivers for executing test 
functions in sequence. 

Traditionally, software testing was done on an ad- 
hoc basis. As the complexity of software increases with 
increasing function being offered, various approaches, 
techniques and tools have been developed to provide a 
more structured or disciplined approach to software 
testing. From a process perspective, typically a phase 
approach is used to divide software testing into different 
stages, such as requirement testing, design testing, unit 
testing, function testing, and system testing. For each 
stage, different testing techniques, such as boundary 
value testing and sampling, are applied to improve test 
coverages. Additionally various testing tools, such as 
test case generators and simulators, are developed to 
improve testing efficiency. 

Traditionally software specification and design 
were also done on an informal basis. Similarly as the 
complexity of software increases with increasing func- 
tion being offered, various techniques and tools have 
been developed to provide a more formal approach to 
software specification and design. Particular examples 
of formal design techniques include top down design 
and data flow analysis. Particular examples of formal 
design tools include formal specification and design lan- 
guages and various Computer Aided Software Engi- 
neering (CASE) tools. 

The advent of formal specification and design lan- 
guages offers a new opportunity for further imposing 
structure and discipline on software testing. The formal 
specifications can be submitted to machine based anal- 
ysis. If the proper behavior of a software interface can 
be adequately described by its formal specification, then 
testing functions can be automatically and systematical- 
ly generated for the software interface from the formal 
specification. Furthermore, the generated testing func- 
tions can be auto-checking. An auto-checking test func- 
tion is a function that can invoke and execute a proce- 
dure to be tested and knows enough about the proce- 
dure to be able to determine whether the procedure be- 
haved properly or improperly when the procedure was 
tested. A particular example of such an auto-checking 
testing function generator is disclosed in US Patent 
5,357,452. 

Typically, each auto-checking testing function re- 
quires a number of parameters to be provided as inputs. 
Each parameter may have one or more attributes, and 
each attribute may take on one of a number of discrete 
values or any value over a range. Thus, even for a small 
number of parameters and attributes, the potential com- 
binations of input values are enormously large, and. for 
all practical purposes, approach infinity. 

Therefore, it is desirable to be able to provide a test 



driver for a software interface that can execute a collec- 
tion of auto-checking test functions selectively to test the 
software interface. It is further desirable if the auto- 
checking test functions can be executed selectively for 
5 selected combinations of the attribute values of the au- 
to-checking test functions' parameter attributes. It is fur- 
ther desirable to have such a test driver automatically 
generated from a specification designating the auto- 
checking test functions of the software interface and 

io specifying selected attribute values of the test functions' 
parameter attributes. A particular example of such a test 
driver generator is disclosed in US Patent 5,359,546. 

Additionally it is further desirable to be able to des- 
ignate the test functions to be tested as a sequence of 

J 5 interrelated test functions. In other words, the values 
produced by one function to be used by another func- 
tion. As will be disclosed, the present invention provides 
methods and apparatus for designating sequences of 
interrelated test functions of software interfaces and 

20 specifying selected attribute values of the test functions' 
parameter attributes, and automatically generating from 
these designations/specifications, test drivers for exe- 
cuting the designated test functions in sequence with all 
combinations of the selected attribute values, thereby 

?5 achieving the desired results described above. 

SUMMARY OF THE INVENTION 

Methods and apparatus for designating sequences 

?o of interrelated test functions of software interfaces and 
specifying selected attribute values of the test functions' 
parameter attributes, and automatically generating from 
these designations/specifications, test drivers for exe- 
cuting the designated test functions in sequence with all 

& combinations of the selected attribute values are dis- 
closed. The methods and apparatus have particular ap- 
plication to automated software testing, in particular, 
software interfaces having a number of procedures and 
corresponding auto-checking test functions that are in- 

'0 terrelated. The procedures and the auto-checking test 
functions may be implemented in any of the well known 
programming languages, such as C, employing any of 
the well known programming methodologies, such as 
objected oriented programming. 

* 5 A test driver generator is provided for generating 
test drivers. The test driver generator receives test ex- 
pressions designating execution sequence of test func- 
tions of software interfaces and corresponding attribute 
value specifications for the designated test functions' 

>° parameter attributes. Each test expression designates 
a number of test functions to be executed in a certain 
sequence, and each corresponding attribute value 
specification specifies selected attribute values of the 
test functions' parameter attributes. For each test ex- 

>s pression and corresponding attribute value specifica- 
tions of a software interface, the test driver generator, 
in response, generates a test driver that can execute the 
specified test functions in the designated order with all 
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combinations of the selected attribute values of the test 
functions' parameter attributes. 

In the presently preferred embodiment, the test 
function sequences and the corresponding selected at- 
tribute values are designated/specified in a combined 
test function sequence and attribute value designation/ 
specification. The test functions are auto-checking. For 
some embodiments, depending on the programming 
languages used for the test functions and the program- 
ming methodologies employed for the software interfac- 
es, the test function sequence designations: and the 
corresponding selected attribute value specifications 
are augmented by supplemental test f unction and/or ob- 
ject type and method definitions. 

In one embodiment, the test driver generator com- 
prises a parser an intermediate representation builder 
and a code generator. The parser receives the test func- 
tion sequence designations and the corresponding se- 
lected attribute value specifications as inputs, and to- 
kenizes the test function sequence designations and the 
attribute value specifications. The intermediate repre- 
sentation builder receives the tokenized test function se- 
quence designations and the tokenized attribute value 
specifications as inputs, and generates intermediate 
representations for these designations/specifications. 
The code generator receives the intermediate represen- 
tations as inputs, and generates test drivers for execut- 
ing the designated test functions in sequence with all 
combinations of the selected attribute values of the test 
functions' parameter attributes. 

Furthermore, in this embodiment, the parser parses 
and tokenizes the test function designations and the at- 
tribute value specifications based on formal grammar. 
The intermediate representation builder builds a syntax 
tree. The code generator generates the test drivers us- 
ing the syntax tree. 

Each test driver is generated with the same execu- 
tion flow for executing a number of test functions in se- 
quence, with all combinations of the selected attribute 
values of the test functions' parameter attributes. For 
each sequence of test functions, the corresponding test 
driver creates the combinations of the selected attribute 
values of the selected test function's parameters one at 
a time. For each combination of the selected attribute 
values, the test driver either creates actual test data in 
line or calls corresponding user supplied test data cre- 
ation functions to create the actual test data. Next, the 
corresponding test driver creates the local test variables 
and evaluate test expressions required for executing the 
designated test functions in sequence. Then the corre- 
sponding test driver invokes the designated test func- 
tions in sequence, and accumulates testing statistics 
based on results returned from the invoked test func- 
tions. After accumulating the statistics, the correspond- 
ing test driver deletes all test data created for the par- 
ticular combination of attribute values, calling user sup- 
plied test data deleting functions if necessary. The proc- 
ess is repeated for each combination of the selected at- 



tribute values of each sequence of test functions. 

For some embodiments, depending on the pro- 
gramming languages used for the test drivers, the test 
drivers are generated with complementary test include 

5 files and/or test driver functions. The complementary 
test driver include files comprise selected attribute value 
definitions for the designated test functions' parameter 
attributes, and user supplied test data creation and de- 
letion function definitions. The complementary test driv- 

io er functions are invoked by the test drivers during exe- 
cution to generate attribute values to create a particular 
combination of attribute values for a test function's pa- 
rameters. 

The present invention will now be further described.. 
75 by way of example, with reference to the accompanying 
drawings, in which: - 

Figure 1 shows a functional block diagram illustrat- 
ing the hardware elements of an exemplary computer 
system that incorporates the teachings of the present 
20 invention. 

Figure 2 shows a functional block diagram illustrat- 
ing the software elements of the exemplary computer 
system of Figure 1 . 

Figure 3 shows a functional block diagram illustrat- 
es ing the input and output of the test driver generator of 
the present invention. 

Figures 4a - 4c illustrate the presently preferred 
embodiment of the combined test function sequence 
and attribute value designation/specification, the sup- 
30 piemental test function definitions and the supplemental 
object type and method definitions of the present inven- 
tion for a software interface. 

Figures 5a - 5c show an exemplary combined test 
function sequence and attribute value designation/ 
35 specification, an exemplary collection of supplemental 
test function definitions and an exemplary collection of 
supplemental object type and method definitions of the 
present invention for an exemplary software interface. 
Figure 6 shows a function block diagram illustrating 
40 one embodiment of the test driver generator of the 
present invention. 

Figure 7 illustrate the execution flow of each gen- 
erated test driver of the present invention. 

Figures 8a - 8c show an exemplary generated test 
45 driver of the present invention. 

Figure 9a and 9b show an exemplary generated test 
driver include file of the present invention. 

Figures 10a-10b-10e showan exemplary collection 
of generated test driver functions of the present inven- 
so tion. 

Figures 11a - 11b show three pairs of exemplary 
user supplied test data creation and deletion functions. 

DETAILED DESCRIPTION 

55 ' 

Methods and apparatus for designated sequences 
of interrelated test functions of software interfaces : and 
specifying selected attribute values for the test func- 
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tions 1 parameter attributes, and automatically generat- 
ing from these designations/specifications test drivers 
for executing the test functions in sequence with all com- 
binations of the selected attribute values are disclosed. 
The methods and apparatus have particular application 
to automated software testing, in particular software in- 
terfaces having a number of procedures and corre- 
sponding test functions, including auto-checking test 
functions. The procedures and corresponding test func- 
tions may be implemented in any of the well known pro- 
gramming languages : such as C, employing any of the 
well known programming methodologies, such as ob- 
jected oriented programming. In the following descrip- 
tion for purposes of explanation, specific numbers, ma- 
terials and configurations are set forth in order to provide 
a thorough understanding of the present invention. 
However, it will be apparent to one skilled in the art that 
the present invention may be practiced without the spe- 
cific details. In other instances, well known systems are 
shown in diagrammatical or block diagram form in order 
not to obscure the present invention unnecessarily. Ad- 
ditionally, while the present invention will be described 
with auto-checking test functions, based on the descrip- 
tions to follow, it will be appreciated that the present in- 
vention may be practiced with test functions that are not 
auto-checking. 

Referring now to Figure 1 , a functional block dia- 
gram illustrating an exemplary computer system that in- 
corporates the teachings of the present invention is 
shown. Shown is a computer 10 comprising a central 
processing unit (CPU) 12, a memory 14, and an I/O 
module 16. Additionally, the computer system 10 also 
comprises an input device 18, an output device 20 and 
a storage device 22. The CPU 12 is coupled to the mem- 
ory 14 and the I/O module 16. The input device 18, the 
output device 20, and the storage device 22 are also 
coupled to the I/O module 16. The I/O module 16 in turn 
is coupled to a network. 

Except for the manner they are used to practice the 
present invention, the CPU 12, the memory 14, the I/O 
module 16, the input device 18, the output device 20, 
and the storage device 22 ; are intended to represent a 
broad category of these elements found in most com- 
puter systems. Their constitutions and basic functions 
are well known and will not be further described here. 

Referring now to Figure 2 a functional block dia- 
gram illustrating the software elements of the computer 
system of Figure 1 is shown. Shown is an operating sys- 
tem 32 comprising a file subsystem 34 and a process 
control subsystem 36. The file subsystem 34 is respon- 
sible for managing files, allocating file spaces, adminis- 
tering free space, controlling access to files and retriev- 
ing data from files. The process control subsystem 36 
is responsible for process synchronization, interprocess 
communication, memory management and process 
scheduling. The file subsystem 34 and the process con- 
trol subsystem 36 are intended to represent a broad cat- 
egory of these elements found in most operating sys- 



tems. Their constitutions and functions are well known 
and will not be further described here. 

The software elements 30 further comprise pro- 
gramming language compilers, software tools/utilities 
5 and their runtime libraries 38, the test driver generator 
of the present invention 40, and the generated test driv- 
ers of the present invention 42. The programming lan- 
guage compilers, software tools/utilities and their runt- 
ime libraries 38 are used to develop and execute appli- 
10 cation programs, in particular, the test driver generator 
40 and the generated test drivers 42 of the present in- 
vention. The language compilers, runtime libraries, 
tools/utilities 38 are intended to represent a broad cat- 
egory of these software elements found in most compu- 
? 5 ter systems. Their constitutions and functions are well 
known, and will not be further described here. The test 
driver generator and the generated test drivers of the 
present invention will be described in further detail be- 
low with references to Figures 3, 6, 7, and 8a - 8c. 
20 While for ease of understanding, the present inven- 
tion is being illustrated with embodiments of the test driv- 
er generator that are implemented in high-level pro- 
gramming languages, and test drivers that ace generat- 
ed in high-level programming languages, it will be ap- 
?5 preciated, based on the descriptions to follow, that the 
present invention may be practiced with the test driver 
generator and the test drivers implemented/generated 
in a variety of programming languages. 

Referring now to Figure 3, a functional block dia- 
io gram illustrating the input and output of the test driver 
generator of the present invention is shown. Shown is 
the test driver generator 40 receiving a number of test 
function sequence designations of software interfaces, 
and corresponding selected attribute value specifica- 
« tions 44 for the test functions' parameter attributes as 
inputs. In the present preferred embodiment, the test 
function sequence designations and the corresponding 
selected attribute value specifications for a software in- 
terface are made in a combined test function sequence 
[ o and attribute value designation/specification. For some 
embodiments, the test function sequence designations 
and attribute value specifications are augmented with 
supplemental test function definitions 46, and/or object 
type and method definitions 48. It will be appreciated 
: 5 that the test function sequence designations, the at- 
tribute value specifications, the supplemental test func- 
tion definitions 46, and the object type and method def- 
initions 48 may be provided to the test driver generator 
40 in a variety of manners. For examples, the designa- 
•o -tions, specifications and supplemental definitions may 
be provided through an input device or pre-stored in a 
storage device accessible to the test driver generator 
40. The test function sequence designations and at- 
tribute value specifications 44, the supplemental test 
5 function definitions 46 and the supplemental object type 
and method definitions 48 will be described in further 
detail with references to Figur s 4a - 4c, and 5a - 5c. 
Continue to refer to Figure 3, for each collection of 
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test function sequence designations and attribute value 
specifications 44 of a software interface, the t st driver 
gen erator 40 generates a test driver 42 for executing the 
designated auto-checking test functions in sequence 
with all combinations of the selected attribute values. 
For some embodiments, depending on the program- 
ming languages used for the test drivers and the pro- 
gramming methodologies employed, the test drivers 42 
are generated with complementary test driver include 
files 50 and complementary test driver functions 52. The 
complementary test driver include files 50 are included 
during compilation of the test drivers 42. The comple- 
mentary test driver functions 50 are invoked by the test 
drivers 42 during execution. The test driver generator 
40 will be described in further detail with references to 
Figure 6. The test drivers 42, the test driver include files 
50 ; and the test driver functions 52 will be described in 
further detail with references to Figures 7, 8a - 8c, 9 
and 10. 

Referring now to Figures 4a - 4c, three diagrams 
illustrating the presently preferred combined test func- 
tion sequence and attribute value designation/specifica- 
tion, the supplemental test function definitions, and the 
supplemental object type and method definitions of the 
present invention are shown. Figure 4a illustrates the 
combined test function sequence and attribute value 
designation/specification, while Figures 4b and 4c illus- 
trate the supplemental test function definitions, and the 
supplemental object type and method definitions re- 
spectively. 

Shown in Figure 4a is a combined test function se- 
quence and attribute value designation/specification 44 
comprising an interface identification 51, a test specifi- 
cation 54, and a number of attribute value specifications 
56 for test function parameters. Test specification 54 
specifies in expression form a sequence of interrelated 
test functions corresponding to procedures of a software 
interface, including their parameters, via one or more 
sub-expressions 53b and/or local variables 53a. In the 
"degenerate case", a test function sequence designa- 
tion may be a single test function specification as in the 
prior art. Each attribute value specification, e.g. 57, 
specifies the selected attribute values for all attributes 
of a test function parameter. Additionally, the attribute 
value specifications 56 may support global declaration 
of attributes and their selected attributes values, thus 
allowing the globally declared attributes and their select- 
ed attribute values to be shared among test function pa- 
rameters. 

Shown in Figure 4b is a collection of supplemental 
test function definitions 46 comprising a number of test 
function definitions, e.g. 58. Each supplemental test 
function definition, e.g. 58, defines an auto-checking 
test function and its parameters. A particular example 
of supplemental test function definition collection is the 
auto-checking test function include file generated by the 
auto-checking test function generator described in U.S. 
Patent 5.357.452. 



Shown in Figure 4c is a collection of supplemental 
object type and method definitions 48 comprising a 
number of object type definitions 60, a number of inher- 
itance definitions 62. and a number of method defini- 

5 tions 64. Each object type definition, e.g. 59 defin s an 
object type of an object oriented software interface. 
Each inheritance definition, e.g. 61 , defines an inherit- 
ance characteristic for an object class and its super 
class. Each method definitions, e.g. 63, defines a meth- 

10 od of an object orient software interlace. A particular ex- 
ample of supplemental object type and method defini- 
tion collection is the object type and method definition 
file generated by the auto-checking test function gener- 
ator described in U.S. Patent 5,357.452 

is Except for the basic requirements described above, 
the auto-checking test functions of a software interface, 
and the corresponding selected attribute values of the 
test functions' parameter attributes may otherwise be 
specified in a variety of manners. For further descrip- 

20 tions of auto-checking test function include files, and ob- 
ject type and method definition files generated by an au- 
tomatic auto-checking test function generator, see U.S. 
Patent 5,357,452. 

Referring now to Figures 5a - 5c, three diagrams 

25 showing an exemplary combined test function se- 
quence and attribute value designation/specification, an 
exemplary collection of supplemental test function def- 
initions, and an exemplary collection of supplemental 
object type and method definitions are shown. Figure 

30 5a shows the exemplary combined test function se- 
quence and attribute value designation/specification, 
while Figures 5b and 5c show the exemplary collection 
of supplemental test function definitions and the exem- 
plary supplemental object type and method definitions 

35 respectively 

Shown in Figure 5a is an exemplary combined test 
function sequence and attribute value designation/ 
specification 44' for an object oriented interface "am" 
comprising an exemplary test function sequence desig- 

40 nation 53' having a sequence of three exemplary inter- 
related test functions , 53a' - 53c\ designating three ex- 
emplary auto-checking test functions, "bank. 
create_account M : "acct. withdraw", and "acct. deposit" : 
their parameters, and local variable "acct" in C like syn- 

45 tax. 

Additionally the exemplary combined test function 
sequence and attribute value designation/specification 
44' comprises three exemplary attribute value specifica- 
tions, 55a* - 55c\ specifying exemplary attribute values 

50 for an exemplary attribute "attrsz" for exemplary test 
function parameters "amt" and "offset" and exemplary 
attribute values for an exemplary attribute "type" for an 
exemplary test function parameter "bank". The exem- 
plary attribute values for the exemplary attribute "type" 

55 55c' are explicitly specified t i.e. "commerce, snal, con- 
sumer cu M . The exemplary attribute values for the ex- 
emplary attribute "attrsz" 55a' and 55b' are implicitly 
specified through an exemplary globally declared at- 
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tribute value "size"., i.e. Viegative : zero, small, large" 57'. 

Note that "acct" is a local variable, created by invo- 
cation of the method "bank.create_accounf, and that 
"offset" is a parameter to the test sequence. Further note 
that the argument to the method "acct. with draw" is con- 
structed by an arithmetic operation combining two pa- 
rameter variables "amt" and "offset" to create a new val- 
ue. 

Shown in Figure 5b is the exemplary supplemental 
collection of test function definitions "amjest.h" 46V 
comprising four exemplary test function definitions 58' 
for four exemplary auto-checking test functions, 
"test_am_create_account", n test_account_deposit", 
"test_account_withdraw", and 
"test_account_get_balance". For each exemplary auto- 
checking test function, the corresponding exemplary 
test function definition specifies its parameters. 

Shown in Figure 5c is an exemplary collection of 
object type and method definitions 48' comprising three 
exemplary object type definitions 60', one exemplary in- 
heritance definition 62\ and four exemplary method def- 
initions 64'. The exemplary method definition "am. 
create_account B resolves the method "bank. 
create_account" (bank = am) to the exemplary auto- 
checking test function "test_am_create_account" 
whose parameters are to be resolved in the exemplary 
supplemental collection, of test function definitions 
"am_test.h". Similarly the exemplary method definition 
"account.deposit" resolves the method "acct.deposit" 
(acct=account) to the exemplary auto-checking test 
function "test_am_account_deposit" whose parameters 
are also to be resolved in the exemplary supplemental 
collection of test function definitions "amjest.h". Like- 
wise, the exemplary method definition "account.with- 
draw" resolves the method "acct. withdraw" (acct=ac- 
count) to the exemplary auto-checking test function 
"test_am_account_withdraw" whose parameters are to 
be resolved in the exemplary supplemental collection of 
test function definitions "amjesth". 

Referring now to Figure 6, a functional block dia- 
gram illustrating one embodiment of the test driver gen- 
erator of the present invention of the present invention 
is shown. Shown is the test driver generator 40 compris- 
ing a parser 66. an intermediate representation builder 
68 ; and. a code generator 70. These elements are se- 
quentially coupled to each other. Together they gener- 
ate the test drivers, the parameter/attribute functions 
and the test driver include files of the present invention 
in C and C like syntaxes respectively for object oriented 
software interfaces whose auto-checking test functions 
are implemented in a programming language that sup- 
ports inter-program call from a C program, in response 
to received test and parameter/attribute specifications 
using C like syntaxes. 

The parser 66. the intermediate representation 
builder 68. and the code generator 70, are implemented 
in like manners similar to a broad category of equivalent 
elements found in many well known programming lan- 



guage compil rs. Their constitutions, basic functions of- 
fered, and operation flows will only be briefly described 
here. 

The parser 66 receives the test function designa- 
s tions and the attribute values specifications, including 
supplemental test function definitions, and/or supple- 
mental object type and method definitions as inputs, and 
tokenizes the various designations, specifications and 
definitions. The intermediate representation builder 68 
10 receives the tokenized designations, specifications, and 
definitions as inputs, and generates intermediate repre- 
sentations for these designations, specifications, and 
definitions. The code generator 70 receives the interme- 
diate representations as inputs, and generates execut- 
'5 able code for the test drivers and the test driver func- 
tions. Additionally, the code generator 70 also generates 
the test driver include files. 

In this particular embodiment, the parser 66 parses 
and tokenizes the expressions based on formal gram- 
me mar. The intermediate representation builder 68 builds 
a syntax tree. The code generator 70 generates the ex- 
ecutable code for the test drivers and the test driver 
functions, and the test driver include files using the syn- 
tax tree. 

25 For further descriptions on various parsers, inter- 
mediate representation builders, code generators, and 
syntax trees, and closures, see A. V. Aho, R. Sethi, and 
J.D. Ullman, Compilers Principles. Techniques and 
Tools, Addison-Wesley 1986, pp. 25 - 388. and 463 - 

30 512. 

It will be appreciated that the test driver generator 
of the present invention may be practiced with other em- 
bodiments having elements providing equivalent func- 
tions provided by the elements of the above embodi- 
es ment. It will further be appreciated that these other em- 
bodiments, may generate test drivers and test driver 
functions in programming languages other than C, re- 
lated outputs in syntaxes other than being C like, in re- 
sponse to test function designations, attribute value 
40 specifications, and supplemental definitions, in syntax- 
es also other than being C like, and/or for software in- 
terfaces that are non-object oriented. 

Referring now to Figure 7. a block diagram illustrat- 
ing the execution flow of each generated test driver is 
45 shown: Each generated test driver has the same exe- 
cution flow 72. Initially the test driver creates a combi- 
nation of attribute values for the test function parame- 
ters, block 73. As described earlier, for some embodi- 
ments, the test driver calls the test driver functions to 
50 generate the attribute values for creating the particular 
combination. 

Then, the test driver creates the test data for the 
particular combination of attribute values, block 74. The 
test driver either creates the test data in line and/or calls 
55 user supplied test data creation functions to create the 
test data. After creating the test data, the test driver in- 
itializes the local variables and evaluates test expres- 
sions required for executing the auto-checking test f unc- 
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tions in sequence, step 75. unless the sequence has no 
local variables nor test expressions, such as the degen- 
erated case of a sequence of one auto-checking test 
function. For such sequences, the test driver proceeds 
directly to step 76. If the test expressions include sub- 
expressions, the sub-expressions are evaluated in or- 
der. At step 76 : the test driver invokes the auto-checking 
test function, block 76. Upon returning from the auto- 
checking test function, the test driver updates the local 
variables and accumulates test statistics based on the 
results returned from the auto-checking test function, 
block 78. After updating the local variables and accu- 
mulating test statistics, the test driver repeats steps 76 
and 78 again, until all auto-checking test functions of the 
sequence have been executed in accordance with the 
designated order. Then, the test driver discards the test 
data created for the particular combination of attribute 
values, block 82, calling user supplied test data deletion 
functions if necessary. 

Blocks 74 - 82 are repeated until they have been 
performed for each combination of the selected attribute 
values specified in the attribute value specifications. 
The user supplied test data creation and deletion func- 
tions will be described in further detail later with refer- 
nces to Figures 11a - 11b. 

Referring now to Figures 8a - 8c, three diagrams 
showing an exemplary generated test driver of the 
present invention are shown. Shown in Figures 8a - 8c 
is the exemplary test driver generated for the exemplary 
test function sequence and attribute value designation/ 
specification and supplemental definitions of Figures 
5a - 5c, having an execution flow as described earlier. 

Initially the exemplary test driver initializes a 
number of control variables, e.g. "totct". The exemplary 
test driver then uses a number of control variables, i.e. 
"I0V "h " and "i2", to loop through execution of the inter- 
related test functions in accordance with the designated 
order. At each pass : the exemplary test driver sets up 
the test variables 86, e.g. "amount am = tv_amt. provide 
(iO)", then executes the interrelated test functions 88, i. 
e. "account acct = acf_create_account(bank, amt); 
acf_withdraw (acct, amt + offset): acf_deposit (acct, 
amt)". As described earlier upon executing the test 
functions for one combination of attribute values, the 
test driver tears down the test data (including the test 
variables) 90, e.g. "tv_bank.relinquish (bank, i2)". The 
remaining source statements illustrated in Figure 8b 
and 8c are for exception handling. 

Referring now to Figure 9, a diagram showing an 
exemplary generated test driver include file is shown. 
The test driver include files are used to define other com- 
mon include files, the attribute values specified, and the 
user supplied test data creation and deletion functions 
for the generated test drivers. 

As shown in Figure 9, the exemplary test driver in- 
clude file 1 02 generated in conjunction with the test driv- 
er for the exemplary test function sequence and attribute 
value designation/specification and the supplemental 



definitions of Figures 5a - 5 comprises four exemplary 
attribute value definitions, 104a - 104d, one for each set 
of the exemplary' attribute values specified, "size", 
"acct_bal". "acct_typ", and "bankjype", and three ex- 

s emplary pairs of user supplied test data creation and de- 
letion functions. "provide_acct" and "relinquish_acct", 
"provide_amt a and "relinquish.amf, and 
"provide_bank" and Velinquish_bank", 106a - 106c. 
Referring now to Figures 10a - 10b, two diagrams 

io showing an exemplary collection of generated test driv- 
er functions are shown. The generated test driver func- 
tions are invoked by the test driver during its execution 
for construction of a particular combination of attribute 
values for the test function's parameters. The generated 

1$ test driver functions comprise a test driver function for 
each of the attributes specified in the test function and 
attribute value designation/specification. Each generat- 
ed test driver function receives an index value as input. 
In response, the generated test driver function returns 

20 an attribute value specified in the test function and at- 
tribute value designation/specification. 

As shown in Figures 10a - 10b, the exemplary test 
driver functions generated for the exemplary, test func- 
tion and attribute value designation/specification and 

25 the supplemental definitions of Figure 5a - 5c, comprise 
four exemplary test driver functions, "size", "acct_bar, 
"acct_typ", and "bank_type", 110-116, one for each of 
the four exemplary attributes "sz°, "baf", "typ" and "type". 
In response to an input value, each of the fou r exemplary 

30 test driver functions, 110, il 2, 114 or 116, returns a cor- 
responding exemplary attribute value specified, i.e. 
"negative", "zero", "small" or "large" for the exemplary 
attribute "size", 110, "neg", "zero", "tiny", "huge" or 
"enorm" for the exemplary attribute "acct_bal", 112, 

35 "economy", "standard", or "gold" for the exemplary at- 
tribute "typ" 114, and "commerce", "snl", "consumer" or 
"cu w for the exemplary attribute "type" 116. 

Referring now to Figures 11a - 11b, two diagrams 
showing three exemplary pairs of user supplied test data 

40 creation and deletion functions of the present invention 
are shown. Each test data creation function creates the 
actual test data for a particular combination of attribute 
values of the test function's parameters. Each test data 
creation function receives the applicable attribute value 

45 for the particular combination as inputs. In response, the 
test data creation function creates the actual test data. 
Similarly each test data deletion function deletes the ac- 
tual test data created for a particular combination of at- 
tribute values. Each test data deletion function receives 

50 the applicable attribute values for the particular combi- 
nation as inputs. In response, the test data deletion func- 
tion deletes the actual test data previously created. 

As shown in Figures 11a - 11b, the exemplary col- 
lection of test data creation and deletion functions, 1 20a 

55 and 120b, comprises three ex mplary pairs of test data 
creation and deletion functions. "provide_acct" and 
"relinquish_acct". 122a and 124a : "provide_amt" and 
"relinquish_amt°, 122b and 124b, and "provide_bank" 
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and u relinquish_bank M ; 1 22c and 1 24c. The 
"provide^acct* and "relinquish_acct" functions. 122a 
and 124a. create and delete an "account" object of a 
particular "account balance". The "provide_amt B and ' 
°relinquish_amf , 1 22b and 1 24b, create and delete an s 
"amount" object of a particular "amount size". The 
"provide_bank" and "relinquish_bank" functions. 122c 
and 124c. create and delete a "bank" object of a partic- 
ular "bank type". 

While the present invention has been described in 10 
terms of presently preferred and alternate embodi- 
ments, those skilled in the art will recognize that the in- 
vention is not limited to the embodiments described. The 
methods and apparatus of the present invention can be 
practiced with modification and alteration within the spir- is 
it and scope of the appended claims. The description is 
thus to be regarded as illustrative instead of limiting on 
the present invention. 



Claims 

1. In a computer system comprising a software inter- 
face having a plurality of procedures and corre- 
sponding test functions for testing said procedures, 2$ 
a method for automatically generating a test driver 
for executing said test functions in sequence, said 
method comprising the steps of: 

a) expressing formally sequential execution or- 30 
der of said test functions; 

b) specifying assignable attribute values se- 
lected for each of said test functions' parameter 
attributes, said test functions comprising a plu- 
rality parameters having a plurality of attributes, 35 
said attributes being assigned attribute values 
when said test functions are invoked: and 

c) generating said test driver based on said for- 
mally expressed sequential execution order of 
said test functions and said specified assigna- 40 
ble attribute values, said generated test driver 
when invoked executes said test functions in 
said designated order with all combinations of 
said specified assignable attribute values. 

45 

2. The method as set forth in claim 1 : wherein, said 
test functions are auto-checking test functions. 

3. The method as set forth in claim 1. wherein, said 
formally expressed test function execution order so 
and said specified attribute value specifications are 
augmented by local test variable declarations and 
supplemental test function definitions. 

4. The method as set forth in claim 1, wherein, said 55 
software interface is object oriented, said formally 
expressed test function execution order and said 
specified attribute value specifications are aug- 



mented by supplem ntal object type and method 
definitions. 

5. The method as set forth in claim 1, wherein, said 
step c) comprises generating an include file for said 
generated test driver, said include file comprising 
attribute definitions and user supplied test data cre- 
ation and deletion function definitions, said include 
file being included at compilation of said generated 
test driver. 

6. The method as set forth in claim 1, wherein, said 
step c) comprises generating test driver functions 
for said generated test driver, said test driver func- 
tions being called by said generated test driver dur- 
ing its execution to provide said generated test driv- 
er with a particular combination of said attribute val- 
ues. 

7. The method as set forth in claim 1, wherein, said 
step c) comprises the steps of: 

c.1) parsing and tokenizing said formally ex- 
j pressed test function execution order and said 
specified attribute value specifications based 
on a formal grammar; 

c.2) generating intermediate representations 
for said formally expressed test function execu- 
tion order and said specified attribute value 
specifications, said intermediate representa- 
tion being a syntax tree; and 
c.3) generating executable code for said test 
driver using said generated intermediate repre- 
sentations. 

8. In a computer system comprising a software inter- 
face having a plurality of procedures and corre- 
sponding test functions for testing said procedures, 
wherein said test functions comprising a plurality 
parameters having a plurality of attributes, said at- 
tributes being assigned attribute values when said 
test functions are invoked, a method for sequential- 
ly executing said test functions in order with pre-se- 
lected combinations of said attribute values, said 
method comprising the steps of: 

a) creating one of said pre-selected combina- 
tions of said attribute values for said test func- 
tions; 

b) creating test data for said created one of said 
pre-selected combinations of said attribute val- 
ues: 

c) evaluating a test expression expressing ex- 
ecution order of said test functions to determine 
a next test function to be executed: 

d) executing said next test functions using said 
created test data; 

e) repeating said steps c) through d) to execute 
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said test functions in accordance with the exe- 
cution order expressed by said test expr ssion: 

f ) deleting said test data created for said creat- 
ed one of said pre-selected combinations of 
said attribute values: and s 

g) repeating said steps a) through f ) for each of 
pre-selected combinations of said attribute val- 
ues. 

9. The method as set forth in claim 8, wherein, 10 

said test functions are auto-checking test func- 
tions: and 

said step d) further comprises accumulating ex- 
ecution statistics for said test functions after ex- 
ecuting each of said test functions. 

10. The method as' set forth in claim 8, wherein, said 
created one of said pre-selected combinations of 
said attribute values is created in said step c) by 20 
calling test driver functions to provide an assignable 
attribute value for each of said attributes of said cho- 
sen test function's parameters. 

11. The method as set forth in claim B, wherein., said 25 
test data is created and deleted in said steps b) and 

f) by calling a user supplied test data creation and 
a user supplied test data deletion function respec- 
tively. 

30 

12. In a computer system comprising a software inter- 
face having a plurality of procedures, a method for 
testing said software interface, said method com- 
prising the steps of: 

35 

a) providing corresponding test functions for 
each of said procedures: 

b) formally expressing a sequential execution' 
order of said test functions, and specifying se- 
lected combinations of attribute values to be 40 
used with said designated order of executing 
said test functions, said selected combinations 

of attribute values being selectively assigned to 
attributes of said test functions' parameter at- 
tributes when said test functions are invoked to 45 
test said procedures in order, said test func- 
tions comprising a plurality of parameters hav- 
ing a plurality of attributes: 

c) generating a test driver that executes said 
test functions in said expressed execution or- so 
der with said selected combinations of said at- 
tribute values, said test driver catling user sup- 
plied test data creation and deletion functions 

to create and delete test data for each of said 
selected combinations of said attribute values: 55 
and 

d) providing said test data creation and deletion 
functions. 



13. The method as set forth in claim 12.. wherein, 

said test functions are auto-checking test func- 
tions: and 

said test driver accumulates test statistics for 
said selected auto-checking test functions after 
executing each of said selected auto-checking 
test functions. 

14. In a computer system comprising a central process- 
ing unit (CPU) and a software interface having a plu- 
rality of procedures and corresponding test func- 
tions for testing said procedures, an apparatus for 
automatically generating a test driver for sequen- 
tially executing said test functions in order, said ap- 
paratus comprising: 

a) designating means comprising said CPU for 
receiving formal expression of execution order 
of said test functions; 

b) specification means comprising said CPU for 
receiving specifications of assignable attribute 
values selected for each of said attributes of 
said test functions' parameter attributes, said 
test functions comprising a plurality parameters 
having a plurality of attributes, said attributes 
being assigned attribute values when said test 
functions are invoked; and ■ 

c) generation means coupled said designation 
and specification means comprising said CPU 
for generating said test driver based on said for- 
mal expression of execution order of test func- 
tions and said specified assignable attribute 
values, said generated test driver when in- 
voked executes said designated test functions 
in accordance with said formally expressed or- 
der of execution, with all combinations of said 
specified assignable attribute values. 

1 5. The apparatus as set forth in claim 1 5, wherein, said 
test functions are auto-checking test functions. 

16. The apparatus as set forth in claim 14, wherein, 

said formal test expressions include local test 
variable declarations; and 
said designation and specification means are 
also for receiving supplemental test function 
definitions to augment some of said test func- 
tion designations and attribute value specifica- 
tions. 

1 7. The apparatus as set forth in claim 1 4 : wherein, said 
software interface is object oriented, and said des- 
ignation and specification means are also for re- 
ceiving supplemental object type and method defi- 
nitions to augment some of said test function des- 
ignations and attribute value specifications. 
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18. The method as set forth in claim 14. wherein, said 
generation means is also for generating an include 
file for said generated test driver, said include file 
comprising attribute definitions and user supplied 
test data creation and deletion function definitions, s 
said include file being included at compilation of 
said generated test driver. 

19. The apparatus as set forth in claim 14 : wherein, said 
generation means is also for generating test driver to 
functions for said generated test driven said test 
driver functions being called by said generated test 
driver during its execution to provide said generated 
test driver with a particular combination of said at- 
tribute values. 15 

20. The apparatus as set forth in claim 14, wherein, said 
generation means comprises: 

c. 1 ) parsing means coupled to said designation 20 
and specification means comprising said CPU 
for parsing and tokenizing elements of said test 
expression and said attribute value specifica- 
tions based on a formal grammar: 
c.2) intermediate representation generation 25 
means coupled to said parsing means compris- 
ing said CPU for generating intermediate rep- 
resentations for said test expression and said 
attribute value specifications, said intermediate 
representation being a syntax tree; and 30 
c.3) code generation means coupled to said in- 
termediate representation generation means 
comprising said CPU for generating executable 
code for said test driver using said generated 
intermediate representations. 35 
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interface am 

size:enum(negative,zero»smalMarge) 



parameter amt=amount ^>55 a i 
attrsz:size 



end 



parameter oflfset=amount ^55^1 
attrsz.size J 



57 



end 



parameter bank=am 1 

attr type:enum(commerce,snal,consumer,cu) fSSc 1 

end 



test local_var acct 

acct = bank.create_account (amth-SSa* 
acctwithdraw (amt and offset); ^-53b» 
acct.deposit ( amt) -v. 53 c i 



FIG. 5a 
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# include "am^ test.h" 

int totct = tv_amt.ctO * t v__of £set . ct ( ) * tv_bank . ct ( ) ; 
int ct=0; 

. report ->predict (totct) ; 

for (int iO = 0; iO < tv_amt.ct(); i0 + + ) { 

for (int il = 0; il < tv_of f set . ct ( ) ; il + ) { 

for (int i2 = 0; i2 < tv_bank . ct ( ) ; i2 + +) { 
Ct + +; 

precond. reset ( ) ; 
report- >start (ct) ; 

try { 

/* test variable set-up */ 
' amount amt « 

t v_amt . provide ( iO ) ; 

precond ->note ( "amt" , amt . toString ( ) ) ; 

amount offset= 
tv_of f set .provide ( il) ; 
J precond- >note ( "of f set " , 

offset . toString () ) ; 
am bank = 

tv_bank .provide (i2) ; 
precond- >note ( "bank" , 

bank . toString < ) ) ; 

report->test_start (ct) ; 



FIG. 8a 
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/* test sequence */ 
account acct = 

ac£_create_account (bank, amt ) ; 

88s acf_withdraw (acct , amt+off set) ; 

acf__deposit (acct , amt ) ; 



report->test_end(ct) ; 
/* test variable tear-down */ 
tv_bank. relinquish (bank, i2) ; 
tv_of fset .relinquish (of fset , il) ; 
tv_amt . relinquish (amt , iO) ; 

} except ex { 

ProvideFail { 

precond->skip() ; 
report->skip (ex. tv, 

ex. index) ; 

} 

RelinquishFail { 

report - > test_error ( "relinquish f ail ' 
ex. tv, ex . index) ,* 

• > 

MissingException ( 

8ubfun->me<ex.before ( ex.after); 

) 

UnexpectedException { 

^ sub£un->ue(ex.before, ex.after, ex.raised); 



FIG. 8b 
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ExceptionallyFault ( 

8ubfun->ef(ex. before, escafter, ex. raised, ex. pad); 

) 

NormallyFault ( 

subfuii->nftex.before , ex. after, ex.pad); 

) 

ExceptionlnCondition ( 

subfun->eic(ex.before, ex. after, ex. raised, 
ex.state, expad); 

{ 

default ( 

panicCbad exception in test driver"); 

) 

\ 

FIG. 8c 
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#pragma once 
Winclude ^ttr^ 
#include m **™ h m 

class attr_bool : public attr ( 
public: 

// this enumeration can be convenient in provide functions, 
enum attr.booLKind (False, True); 
char*print(); 
int val; 

attr JxuKint i); 
static int ct(); 

); 

class attr_size : public attr ( 
public: 

// this enumeration can be convenient in provide functions* 
enum attr_size_Kind {negative, zero, small, large); 
char*print(); 
int val; 

at±r_size(int i); 
static int ctO; 

); 

class attr_acctj)al : public attr ( 
public: 

// this enumeration can be convenient in provide functions, 
enum attr_acct_bal.J£ind (neg, zero, tiny, huge, enortn); 
char*print(); 
int val; 

attr_acct_bal(int i); 
static int ct(); 



>l04a 



I02 



H04b 



); 



FIG. 9a 
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class attr_acct_typ : public attr { 
public: 

// this enumeration can be convenient in provide functions, 
enum attr_acct_typ_Kind (economy, standard, gold) ; 
char*print(); 
int val; 

attr_acctjyp(int i); 
static int ctO; 

); 

class attr_bank_type : public attr ( 
public: 

// this enumeration can be convenient in provide functions, 
enum attr_bank.type.Kind (commerce, snl, consumer, cu); 
char*print(); 
int val; 

attr_bank_type(int i); 
static int ct(); 

); 

// the following must be defined by the user 

bool provide_acct (am.accountjEp &acct_var, attr_acct_bal bO ( 

attr_acct_typ tO); 
void relinquish_acct (am_account_fp &acct_var, attr_acct_bal bO, 

attr_acct_typ tO); 



>l04c 



>l04d 



H06a 



30); J 



106b 



bool provide_amt (int &amt_var, attr_size sO); 
void relinquish_amt (int &amt_var, attr_size 

bool prt>vide_bank (am_fp S&bank_var, attr_bank_type tl); H06c 
void relinquish_bank (am_fp &bank_var, attrj>ank_type tl); J 

FIG. 9b 
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#include "am_attr.h" 
char* attr_bool:rprintO ( 
switch (val) { 

case False: return "False"; 
case True: return True"; 
default: panicC'unknown attr val"); 
return "????"; 

) 

) 

attr_bool::attrj)ool(int i) { 
if (i < 0 I I i > ct ()) { 

panic ("bad value to attr constructor"); 

val » i; 

} 

int attr_booI::ct () { 
return 2; 

) 



char* attr_size::print () { 
switch (val) { 

case negative: return "negative"; 
case zero: return "zero"; 

case small: return "small"; 
case large: return "large"; 
default: panic ("unknown attr val"); 
return "???? H ; 

} 

) >IIO 

attr_size::attr size (int i) { 
if (i <0 ll i > ct()) { 

panic ("bad value to attr constructor"); 

) 

val = j 



int attr_size::ct () ( 
return 4; 

) 



FIG. IOa 
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char* attr_acct_bal::print () { 
switch (val) ( 
case neg". 
case zero: 
case tiny: 
case huge: 
case enorm: 



) 



return neg ; 
return "zero"; 
return "tiny"; 
return "huge"; 
return "enorm"; 
default: panic ("un know n attr val"); 
return "TTT?"; 



} 

attr_acctj>al::attr__acctj)al(int i) { 
if (i < 0 I I i > ct ()) ( 

panic("bad value to attr constructor") 

) 

val = i; 

) 

int attr_acct_bal::ct() ( 
return 5; 

) 



>II2 



FIG. 10b 
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char* attr_acct_typ::print() { 
switch (val) ( 

case economy: 
case standard: 
case gold: 



return "economy"; 
return "standard"; 
return "gold"; 



) 



default: panic ("unknown attr val"); 
return "????"; 



) 

attr_acct_typ:: attr_acct_typ (int i) { 
if (i < 0 I I i > ct ()) ( 

panic ("bad value to attr constructor"); 

val = i; 

) 

int attr_acct_typ::ct() ( , 
return 3; 

} ' 

char* attr_bank_type::print() ( 
switch (val) ( 

case commerce: return "commerce"; 
case snl: return "snl"; 

case consumer: return "consumer"; 

case cu: return "cu"; 

default: panic ("unknown attr val"); 
return "????"; 



>II4 



108c 



) 
) 

attr_bank_type::attr_bank_type(int i) { 
if (i < 0 I I i>ct())( 

panic ("bad value to attr constructor"); 

) 

val = i; 

) 

int attr_bank_type::ctO ( 
return 4; 

) 



>II6 



FIG 10c 
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♦include *amJT 
tindude 'tm.tttr.h" 



const char •aervice_najne « *iunple_aecount w manAgor r ; 



boo] 

provide.acct Caxn.accountjp &a, attr_acct_baJ baJ) ( 

am Jp acct.mgr = EZLOOKUP (*vuJa£e_namer, service^naunie, am); 

ELDQ_jLtxi0uxit openings balance; 
switch (baJ.vml) ( 

case attr fc acct_baJ::neg: 

epenia£_baJanca «= -10; 

break; 
case attr.acct.bal:: zero: 

opening_balance « 0; 

break; 
case »ttr_acctj»l::tiny; 

opening_b alanee ■ 100; 

break; 
ate attr_acc tJjaJL : huge : 

openin£_balance = 1000; 

break; 

ease attr_a«t_baJ:;e aorta; 

opetung_balaiice a 10000; 
break; 

default: 

// ehould not" get here 

cout « "bad value for attr_acct..bal\n"; 

return FALSE; 

try ( 

a « acct_ nogr->create_a ceo unt(openin ^balance); 
) except ex ( 
am_negative_amount ( 

cout « "create_accour»t raised am. negative amountW; 
return FALSE; 

1 

contract, fault { 
cout « 

fora("create_account raised contract Jailure (*%x)\n", ex.code); 
return FALSE; 

I 

default ( 

cout « "create_account raised unknown exception": 
return FALSE; 

I 

) 

return TRUE; 



> 122a 



120a 



void 

reUnquiih_accUarD_account Jp &a, attr_acct__baJ bad) ( )> 124a 
a->consume(); 



FIG lla 
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bool 

provide.aznt (am.amount &a, attr.size size) [ 

Ewitch(size.val) ( 

case attr_size::negative: 

a = -10; 

break; 
case attr_8ize::;ten>: 

a=0; 

break; 
case attr size::small: 

a =100; 

break; 
case attr_size::large: 

a » 1000; 

break; 

default: 

// should not get here 

cout « "bad value for attr_size_sizeW 

return FALSE; 

) 

return TRUE; 



>l22b 



120b 



void 

relinquish.amt (airramount &a ( attr_size a) ( 
// no work to do 

i 



} 



124b 



bool 

provide_bank (am.fp &a, attr J)ank_type tO) ( 
if (tO.vaJ - attr_bank-type -consumer) ( 

am.fp acct.mgr = EZLOOKUP (*village_namer, 
service_name f am); 
a = acct_ mgr; 
return TRUE; 
)else ( 
return FALSE; 

) 



>l22c 



void 

relinquieh_bank (am_fp &bank_var, attr.bank.typc tO) 
// make bank_var go away, the test driver is done with 
bank_var->consumer<); 

FIG. lib 



4 



124c 
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